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HIGHLIGHTS: 

■ Intervention : Small public high schools in New York City, created citywide in mostly high-poverty 
communities to replace large, low-performing high schools. The small schools compete for students through 
the city’s system of school choice. 

■ Evaluation Methods : A large, well-conducted randomized controlled trial. 

* Key Findings : Four years after random assignment, a 6-10 percentage point increase in the four-year high 
school graduation rate, and a 4-6 percentage point increase in the rate of four-year graduation with a New 
York State Regents diploma (requiring proficiency on each of five state Regents exams in various subjects). 

■ Other : 

(i) The small schools’ operating cost is approximately the same as that of the larger, more traditional high 
schools attended by control group students. 

(ii) A study limitation is that it evaluated the intervention in a single city that was concurrently 
implementing other system-wide education reforms. Replication of these findings in a second trial, in 
another setting and different conditions, woidd be desirable to confirm the initial results and establish 
that they generalize to other situations where the intervention might be implemented. 


I. The Top Tier initiative’s Expert Panel has identified this intervention as Near Top Tier. 

The Panel finds that this intervention meets the “Near Top Tier” evidence standard, defined as: 

Interventions shown to meet almost all elements of the Top Tier standard (i.e., well-conducted 
randomized controlled trials... showing sizable, sustained effects), and which only need one 
additional step to qualify. This category includes, for example, interventions that meet all elements of 
the standard in a single site, and just need a replication trial to confirm the initial findings and 
establish that they generalize to other sites. 

II. Description of the Intervention: 

Between 2002-2008, New York City created Small Schools of Choice (SSCs) to replace large public 
high schools with graduation rates below 45 percent located in disadvantaged communities. SSCs are 
small (roughly 100-120 students per grade, as compared to the usual 350 or more in traditional city 
high schools), academically nonselective, and designed to ensure students receive individualized 
attention from teachers. The schools were newly created through a competitive process, in which the 
city invited applications from prospective school leadership teams. 

Additional SSC features include: (i) new principals and teachers (as opposed to transfers from a large 
high school that the SSC replaced); (ii) start-up funding from the city’s Department of Education and 
philanthropic organizations (such as the Bill and Melinda Gates Foundation); (iii) assistance with 
leadership development, staff hiring, and program start-up from intermediary organizations (such as 
New Visions for Public Schools); and (iv) partnerships with local businesses or nonprofit 
organizations that offer students learning opportunities inside and outside the classroom. 

SSCs operate in the context of the city’s high school choice system, under which all rising 9 th graders 
ra nk order their preferences for high schools to attend, and then are placed in their most preferred 
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school with an available spot. Researchers call these high schools “small schools of choice” because 
they are small and they are open to all students who choose them regardless of the students’ past 
academic performance. 

SSCs’ operating cost is about $58,000 per student over five years (in 2014 dollars), which the study 
described below found to be approximately the same as that of the larger, more traditional high 
schools attended by students in the control group. 1 

III. Evidence of Effectiveness: 

This summary of the evidence is based on a systematic search of the literature to identify all well- 
conducted randomized controlled trials of SSCs. Our search identified one such trial. What follows is 
a summary of the study design and the intervention’s effects on the main outcomes measured in the 
study, including any such outcomes for which no or adverse effects were found. All effects shown are 
statistically significant at the 0.05 level unless stated otherwise. 

Overview of the Study Design : Randomized controlled trial with a sample of 18,000 rising 9 th 
graders who applied to one of 105 over-subscribed SSCs in New York City during 2005-2008. 

The study sample comprised approximately 18,000 students who applied to enter 9 th grade in one 
of 105 oversubscribed SSCs in the city during 2005-2008 (out of a total of 123 SSCs created 
during these years). These students were randomly assigned via lottery to (i) a group that was 
eligible to attend the SSC, or (ii) a control group that was not. 2 

The study estimated the effects of SSCs using the sample of 14,969 students who applied to SSCs 
during the first three of the four study years - i.e., 2005, 2006, and 2007 - since these were the 
students for whom four-year outcome data were available for the most recent study report. 3 46% 
of students in this sample were Hispanic, 44% were African American, 70% performed below 
grade level in reading and 68% did so in math, and 84% were eligible for free or reduced-priced 
school lunches due to low family income. 


1 This cost estimate is based on the sample of students who applied to SSCs during the first two study years, for 
whom five-year outcomes were available. One reason SSCs did not cost more than the schools attended by control 
group students is that a greater percentage of SSC students graduated in four years, and thus fewer stayed in high 
school for a fifth year requiring an additional year of school expenditures. 

The cost estimate for SSCs does not include the one-time start-up costs of creating an SSC, the ongoing costs of 
facility usage (e.g., gyms, science labs), nor the cost of resources contributed to SSCs by external partners to support 
ongoing operations. However, a careful analysis found that these factors are unlikely to change the cost estimate 
substantially because (i) the one-time start-up costs are relatively small when amortized over many years; (ii) the 
difference in facility usage costs between SSCs and other schools is minimal; and (iii) external partners’ funding 
contributions for ongoing operations are relatively small, based on a survey of such partners conducted by New 
Visions for Public Schools (Bifulco, Unterman, and Bloom 2014). 

2 Some of the students who lost the lottery for their first-choice SSC ended up attending an SSC that was not their 
first choice (e.g., through a subsequent stage of the lottery process). The study appropriately kept these “cross-over” 
students in the control group when estimating SSC’s effects, consistent with an intention-to-treat approach. As 
described below, the study also presents “treatment on treated” effects that adjust for these cross-overs. 

3 A forthcoming study report will present four-year outcomes for the hill sample, as well as longer-term outcomes 
for students who entered the study during its initial years. A preliminary brief on these findings (Unterman 2014) 
shows similar effects on high school graduation to those presented here, and positive effects on college enrollment. 
We plan to revise this summary to include the new results once they are fully reported. 
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Effects of SSCs four years after random assignment, compared to the control group : 


■ A 6 percentage point increase in the four-year high school graduation rate (69.6% of the SSC 
group graduated from high school in four years, versus 63.6% of the control group). 

■ A 4 percentage point increase in the rate of four-year graduation with a New York State 
Regents diploma, requiring proficiency on each of five state Regents exams in various 
subjects (44.9% of the SSC group versus 41.3% of the control group). 

* A 4 percentage point increase in the rate of students scoring high enough on the Regents 
exam in English to be considered college ready by the City University of New York (CUNY) 
system (39.5% of the SSC group versus 35.4% of the control group). 

* No significant effect on the percent of students graduating in four years with an Advanced 
Regents diploma (awarded to exemplary students), or on the percent scoring high enough on 
the Regents math exam to be considered college ready in math by the CUNY system. 4 

The effects of winning the SSC lottery, shown above, likely understate the effects of actually 
attending an SSC, because some of the students who won the lottery did not attend an SSC, and 
some students who lost the lottery attended an SSC anyway (e.g., by winning a second-stage 
lottery). The study estimates that the effects of actually attending an SSC (the “treatment on 
treated” effects) are greater than shown above - namely, a 1 0 percentage point increase in the 
four-year high school graduation rate, a 6 percentage point increase in the rate of four-year 
graduation with a Regents diploma, and a 7 percentage point increase in the college -readiness rate 
in English. These effects may be valid, but confidence in them is partially tempered by their 
reliance on an analytical assumption that cannot be directly verified. ' 

Discussion of study quality : 

■ The study had low sample attrition and a reasonably long-term follow-up: Outcome data were 
obtained for 8 1 % of the SSC group and 8 1 % of the control group, four years after random 
assignment. 

* The SSC and control group students in both the original randomized sample and the four-year 
follow-up sample were highly similar in their observable pre -program characteristics (e.g., 
demographics, academic achievement). 

■ The study appropriately kept sample members in the groups to which they were randomly assigned, 
when analyzing the intervention’s effects (i.e., the study used an “intention-to-treat” analysis). 

■ All study outcomes were measured using official administrative records from the New York 
City Department of Education. 


4 The Regents English and math exams are the two Regents exams that the CUNY system uses to determine whether 
a student has met its proficiency requirements for enrollment in a bachelor’s degree program. 

5 An assumption underlying these “treatment-on-treated” effects is that the intervention’s effect on each student in the 
control group who attended an SSC (i.e., each control group “cross-over” student) is the same as the intervention’s 
effect on those students who were randomly assigned to the same SSC (i.e., the treatment group students in the same 
school). This assumption can neither be verified nor disproved. 
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■ The study evaluated SSCs as delivered on a very large scale in New York City, thus providing 
evidence of the intervention’s effectiveness under real world implementation conditions. 

■ Study limitation : The study evaluated SSCs as implemented in a single city that was 
concurrently implementing other system-wide education reforms (e.g., centralization of 
school district governance under the mayor, changes in teacher hiring policies to de- 
emphasize seniority). The Top Tier initiative’s Expert Panel believes that replication of the 
above findings in a second trial, conducted in another setting and different conditions, would 
be desirable to confirm the initial results and establish that they generalize to other situations 
where SSCs might be implemented. 

IV. Summary of the Intervention’s Benefits and Costs: 

If taxpayers fund the delivery of this intervention, what benefits to society can they expect to result, 
and what would be their net cost? The following table provides a summary. This is intended to be a 
general overview of social benefits in relation to taxpayer cost, rather than a comprehensive benefit- 
cost analysis. It assigns monetary value to particular benefits and costs only when doing so requires 
minimal assumptions. The monetary amounts shown are in 2014 dollars. 


Benefits To Society 

Four years after random assignment: 

■ A 6-10 percentage point increase in the four-year high school graduation rate. 

■ A 4-6 percentage point increase in the rate of four-year graduation with a New York State Regents 
diploma (requiring proficiency on each of five state Regents exams in various subjects). 


Cost To Taxpayers 

SSCs’ operating cost is about $58,000 per student over five years, which is approximately the same 
as that of the larger, more traditional high schools attended by control group students. 
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review of this intervention. 


5 


